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ABSTRACT 



The feasibility of using a computer-based networked 
collaborative concept mapping system to measure teamwork skills was studied. 

A concept map is a node -link -node representation of content, where the nodes 
represent concepts and links represent relationships between connected 
concepts. Teamwork processes were examined for a group concept mapping 
activity using networked computers and HyperCard software developed for the 
study. Two pilot studies investigated the feasibility of the approach, and 
one of these is reported in this study. Thirty 9th graders worked together in 
the first pilot study to create collaborate maps on the computer systems. 
Results of this study and a second confirmatory study resulted in development 
of a study with 23 groups of 3 students each from middle schools and high 
schools. Usable data were recorded for 15 groups (45 participants) . Results 
suggest that the real-time measurement of teamwork skills is feasible and 
that students using the networked computers were able to create a concept map 
jointly. The number of concepts used by groups ranged from 7 to 18, and the 
number of links ranged from 8 to 26. It is suggested, however, that the task 
involved may be more like a small group task than a "team" task if the 
purpose of a team is defined as performing a task an individual cannot 
perform alone. The developed software architecture has been designed to be 
domain independent and should be useful in other computer-based team 
environments. (Contains 1 figure, 8 tables, and 20 references.) (SLD) 
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Use of Networked Collaborative Concept Mapping 
to Measure Team Processes and Team Outcomes 

In this study we examined the feasibility of using a computer-based, networked 
collaborative concept mapping system to measure teamwork skills. A particularly novel feature of 
our work is that we are refining an approach that employs networked computers to capture and 
measure - in real-time - team processes for individual students and teams. The current work, 
together with our past efforts (e.g., O’Neil, Allred, & Dennis, in-press-a; O’Neil, Chung, & 
Brown, in press-b), suggests a promising approach toward the development of a real-time 
measurement system for teamwork skills. 

Teamwork Processes 

Our work focuses on assessing team processes used by a group of individuals responsible 
for jointing constructing a concept map. A concept map is a node-link-node representation of 
content, where nodes represent concepts and links represent relationships between connected 
concepts (Dansereau, 1995; Jonassen, Beissner, & Yacci, 1993). We are interested in measuring 
the interaction between team members as they work together to construct a group concept map. 

Our definition of team is from Tannenbaum, Salas, and Cannon-Bowers’ (1996) and Baker 
and Salas (1992): “... two or more people who interact dynamically, interdependently and 
adaptively and who share at least some common goals or purpose,” and where the team members 
have built-in dependencies and where each member has well defined roles and positions. 

We have drawn on the work of Morgan, Salas, and Glickman (1993), Salas, Dickinson, 
Converse, and Tannenbaum (1992), Burke, Volpe, Cannon-Bowers, and Salas (1993), and others 
(O’Neil, Baker, & Kazlauskas, 1992; Webb, 1993; Webb & Palincsar, 1996) to aid our 
development of teamwork process measures. Morgan et al. provide insight into the nature of 
teams. In their model of team development, Morgan et al. postulate two tracks of team processes, 
a taskwork track and a teamwork track. The taskwork track accounts for specific activities unique 
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to the task. Taskwork team skills influence how well a team performs on a particular task (e.g., 
how well a team of students are able to jointly construct a concept map). Taskwork skills are 
domain-dependent, task-related activities. The teamwork track or teamwork skills influence how 
effective an individual member will be as part of a team and are domain-independent. Teamwork 
skills encompass skills such as adaptability, coordination, cooperation, and communication. 
Effective teams develop competence along both tracks. Members of effective teams possess basic 
skills required for the task and know how to coordinate their activities, communicate with each 
other, and respond effectively to changing conditions. 

In the current work we use the taxonomy of teamwork process developed by O’Neil et al. 
(in press-b). The taxonomy is made up of six processes: (a) adaptability - recognizing problems 
and responding appropriately, (b) communication - the exchange of clear and accurate information, 
(c) coordination - organizing team activities to complete a task on time, (d) decision-making - 
using available information to make decisions, (e) interpersonal - interacting cooperatively with 
other team members, and (f) leadership - providing structure and direction for the team. Our prior 
research showed statistically significant positive relationships between decision making and team 
performance on a negotiation task (final contract offer and quality of agreement), and a significant 
negative correlation between interpersonal processes and whether an agreement was reached 
(O’Neil et al., in press-b). 

Team processes in a group concept mapping. In this study we examined teamwork 
processes in the context of a group concept mapping activity, where communication occurred over 
networked computers. To the best of our knowledge, this has never been done before - we know 
of no theoretical framework that has been tested using a group concept mapping task. Thus, our 
expectations about what processes team members invoke are tentative at best. 

We view group concept mapping as a weak instance of Tannenbaum et al.’s (1996) definition 
of a team, particularly in member interdependency and specialized member function. We expect 
member interdependency to be low given the ease of concept mapping. We have found through 
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pilot testing that elementary school children to graduate students can successfully use our system to 
construct concept maps. The ease of use of our system and the simplicity of concept mapping 
means that anyone can carry out the task, which means that there are no specialized skills necessary 
to carry out the task. Given our typical team scenario of (a) teams comprised of non-experts and 
(b) one leader and two advisory members (see O’Neil et al., in press-b for a detailed description), 
and the simplicity of the task, we expect each person on a team (even novices) to be able to 
perform either role equally well. Finally, we expect concept mapping performance to be essentially 
knowledge driven. To the extent that teamwork processes emerge, we expect these processes in 
general to be mediated by the content expertise of team members. Teams with members low in 
content knowledge are expected to focus primarily on surface features of the task and far less on 
the substantive part (Chi, Glaser, & Farr, 1988). 

Thus, given these characteristics of group concept mapping - easy to use computer system, 
easy to complete task, content novices - we expect low performance and we do not in general 
expect to see strong relationships between many of the team processes and concept map 
performance. The one exception is decision making. Given that concept maps are knowledge 
dependent, we expect teams that engage in discussion about the content to make reasoned, 
informed decisions about concepts and their relations, and thus perform better on the concept 
mapping task. 

Measuring Team Processes Within a Networked Environment 

Existing approaches to measuring teamwork processes almost exclusively rely on 
observational methods (Baker & Salas, 1992). For example, behavioral checklists (e.g., Oser, 
McCallum, Salas, & Morgan, 1989), videotaped and audiotaped observation (e.g., Brannick, 
Roach, & Salas, 1993), and analysis of think-aloud protocols are the most common techniques to 
measure teamwork processes. These methods are labor intensive and time-consuming. 
Observations must be transcribed, coded, and analyzed post hoc. Such techniques offer no 
opportunity for rapid analysis and reporting of team performance. From an assessment 
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perspective, these methods are unappealing because of the lag between test administration and 
reporting of test results. Further, these methods are neither practical nor cost-effective in large- 
scale test settings. Our approach to addressing these limitations is to employ computer-based 
assessments. Our prior work suggests that this is a feasible and promising approach to assessing 
teamwork skills (O’Neil et al., in press-b). The work reported here is an extension of our 
approach to a different domain and a different task. 

To measure teamwork skills we use a domain independent taxonomy of teamwork processes 
to guide the development of a set of domain specific messages. These predefined messages are the 
sole means for members to communicate with each other, and the messages map on to the 
teamwork processes (adaptability, coordination, decision making, interpersonal, and leadership). 
As the team carries out the task the software tracks which messages (and hence which categories) 
are used by each member. Thus, message usage provides an index of the kinds teamwork 
processes members are using. We assume that each message in a category is as important as any 
other message, and thus all messages are equally weighed. This technique provides us with a real- 
time teamwork assessment system. The assessment can be administered, scored, and reported in 
real-time. 



PILOT STUDIES 

Two pilot studies were conducted to assess the feasibility of our approach. The first pilot 
study was our initial attempt at assessing the functionality of the computer system and determining 
the kinds of messages participants (in a group) use to construct a concept map. We were interested 
in feedback from users regarding usage of the system and the task. The second pilot study 
reflected several major revisions to the system based on the first pilot study, and verified that our 
technique was feasible (i.e., students could jointly construct concept maps using our system). 

Only the first pilot study is described here. 
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Pilot Study 1 

Participants and team concept mapping task. 30 ninth grade male and female 
students were randomly drawn from five lOth-grade classes. All students spoke English as their 
first language. 

We assigned participants to three-person teams where each person had their own computer 
and the computers were networked together. The scenario we used was one where there was one 
leader and two members. Only the leader could manipulate the concept map (e.g., adding concepts 
and links). Non-leaders could only advise by sending messages. 

Participants were instructed to jointly construct a concept map on environmental science. 
One of the reasons we used environmental science as the topic was that it was part of the 
curriculum; thus, students were presumed to have some familiarity with the subject. We provided 
the team with 15 terms (atmosphere, bacteria, climate, C02, decomposition, evaporation, food 
resources, oxygen, precipitation, respiration, sun light, vegetation, photosynthesis, waste, water) 
and 9 links (contributes to, causes, leads to, part of, result of, similar to, produced by, influences, 
type of, prior to). 

Leadership passed every 12 minutes (i.e., 1/3 of the task time) among team members to 
allow each member to control the map once. Teams were given 36 minutes to complete the concept 
map. The map was scored in real-time by the computer using a scoring algorithm that compared 
students’ maps with an expert’s map (Herl, Baker, & Niemi, 1996). 

Networked concept mapper. The system was developed on the Macintosh® with 
HyperCard®. We used the built-in networking capabilities of the operating system (System 7®) 

and HyperCard® to implement a rudimentary peer-to-peer system. Every member’s screen was 
updated as changes occurred (e.g., someone sending a message, or the leader making changes to 
the concept map); thus, all computers were synchronized with each other. Communicating 
between team members occurred via typed messages. To send a message, members typed what 
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they wanted to say in a special message box and then sent the message. The message was 
dispatched to other members’ computers and appeared in the order they were sent. To make 
changes to the concept map, the leader could add, delete, or move concept map components around 
the screen. Each time a concept map event occurred, the event was dispatched to other members’ 
computers and their concept maps automatically updated. All typed messages and concept map 
events were logged by the computer. 

Results. All groups participated in the concept mapping task with sustained interest and 
effort. There was unanimous agreement from all participants and their teachers that the 
collaborative mapping task was interesting, engaging, and fun. All groups, with only a few 
minutes of instructions, were able to immediately use the system and engage in the task. All 
groups were able to construct a map within the allotted time. 

Team processes. 798 messages were sent by all groups. All messages were rated by 
one experimenter. Messages were rated first as on-task or off-task. 557 messages were rated as 
on-task (74%) and 211 (26%) messages were rated as off-task. On-task was defined as being 
directly related to constructing a concept map (e.g., “Add evaporation and make a link to water.” 
versus “What was the last good movie you guys saw?”). On-task messages were then categorized 
into one of the teamwork process categories. Total message usage was: adaptability - 151 
messages (26%), coordination - 12 messages (2%), decision making - 80 messages (14%), 
interpersonal - 15 messages (3%), leadership - 223 messages (38%), and communication - 87 
messages (15%). Two additional categories were used to capture messages that reflected errors 
with using the system (e.g., a message reflecting a typing error, 3 messages, 1%) and a second 
category to record uncategorizable messages (16 messages, 3%). 

Team concept mapping performance. Team performance was measured with 
concept mapping scores. The concept maps were scored by comparing the group map with an 
expert map in terms of the content and structure of the map (Herl et al., 1996). Herl et al. reported 
that this concept mapping scoring approach yielded reliable scores with strong positive correlations 
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with other measures of content knowledge (e.g., essay writing and prior knowledge short answer 
questions). In this pilot study, the average team content score was 6.4 ( SD=2.12 , range=2 to 11) 
out of a possible 38. The average team structure score was .29 (SD=. 08, range=.14 to .37) out of 
a possible 1 .0. The scores indicate little content knowledge of these students compared to an 
expert. 

Individual message counts were also scaled by the total number of messages sent by the 
group. Nonparametric (Spearman) correlations were then examined between the six teamwork 
processes and the two outcome measures. For the content score, a significant negative correlation 
was found for communications (ry(10)=-.74,p=.007). The more teams used communication 
messages, the lower they scored on the content measure. For the structure score, a significant 
positive correlation was found for decision making (ry(10)=.59,p=.02), and a significant negative 
correlation for adaptability (r 5 (10)=-.65,p=.02). The more teams used decision making messages 
(e.g., “Why oxygen [sic] why not a natural disaster?” ), the higher they scored on the structural 
measure. Conversely, groups that used more adaptability messages (e.g., “What should we link 
next?”) tended to score significantly lower on the structural measure. 

These results, while limited in scope, are consistent with our previous findings (O’Neil et 
al., in press-b). In particular, teams could use predefined messages to communicate with each 
other, and teams could complete the task. Also, decision making seems to be a key factor in how a 
group performs. Groups that engaged in higher levels of decision making tended to perform 
higher on the concept mapping task. Also consistent with O’Neil et al. was that no other 
significant correlations were found between other group processes and team performance. 
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MAIN STUDY 
Method 

Participants 

Twenty-three groups (69 participants) were randomly drawn from four middle school and 
high schools to participate in this study. All students spoke English as their first language. 
Technical problems resulted in some computers crashing, which resulting in usable data for only 
15 groups (45 participants). 

Networked Concept Mapping System 

Table 1 lists the specifications for our networked concept mapping system. Our system 

was developed using HyperCard® running on the Macintosh®. The software was designed to be 
simple and easy to use. Participants sent messages by clicking on numbered buttons. Concepts 
were added to the concept map via menu selections, and links were created by connecting two 
concepts and then selecting the desired link from a pop-up menu. Our pilot studies and in-house 
usability testing showed that participants of various ages (5th graders to graduate students) could 
be trained to use the system within a few minutes. 
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Table 1 . Domain Specifications Embedded in the Software 



General domain specification 


This software 


Scenario 


Create a concept map on environmental science by 
exchanging messages in a collaborative context 


Participants 


Student team (3 members) 


Concept map terms 


Predefined. 18 important ideas identified by content 
experts: atmosphere, bacteria, carbon dioxide, climate, 
consumer, decomposition, evaporation, food chain, 
greenhouse gases, nutrients, oceans, oxygen, 
photosynthesis, producer, respiration, sunlight, waste, and 
water cycle. 


Concept map links 


Predefined. 7 important relationships identified by content 
experts: causes, influences, part of, produces, requires, 
used for, and uses 


Type of learning 


Content understanding 


Outcome measures 


Semantic content scores (stringent and categorized), 
organizational structure score, number of terms used, 
number of links used. 


Teamwork processes 


Adaptability, coordination, decision making, interpersonal, 
leadership, communication 



User Interface. Figure 1 shows the user interface to the system. The display was 
partitioned into three major sections. The top- fifth was reserved for concept mapping. The lower 
left of the screen displayed the messages. All messages sent were listed in the order sent by 
members. Thirty-five numbered buttons were provided to members. Each participant was given a 
paper handout that listed all messages, and the messages were numbered to correspond to the 
buttons on the computer screen. To send a message, participants clicked on a button and the 
corresponding message was sent to everyone’s computers. Other information shown on the lower 
part of the screen was the remaining time for the entire task, and the remaining time as a leader or 
non-leader. As shown in Figure 1 , the remaining time as a leader (i.e., person who controls the 
map) is two minutes, and the time remaining for the entire task is 32 minutes. 
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Figure 7. User interface for the system. 



Messages. In order to capture team processes in real-time, we provided participants with 
37 messages that belonged to one of the five team process categories. Team members used these 
messages to communicate with each other during the concept mapping task. 



Each message belonged to one of the five teamwork process categories (adaptability - 7 
messages, coordination - 7 messages, decision making - 6 messages, interpersonal - 7 messages, 
and leadership - 7 messages). Two messages (“Yes” and “No”) were part of the communication 
category. Another message was provided to signal that a previously sent message was mistakenly 
sent by the participant. This message was excluded from the message categories. A sample 
message from each category is given in Table 2. 
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Table 2. Definitions and example messages for each team process category. 



Team process 


Definition and example 


Adaptability 


Recognizing problems and responding appropriately. [What do you think, 
Ml? (Ml refers to Member 1 of the team.)] 


Coordination 


Organizing team activities to complete a task on time. (We only have 5 minutes 
left.) 


Decision making 


Using available information make decisions. (What if we add ‘carbon 
dioxide’?) 


Interpersonal 


Interacting cooperatively with other team members. (1 need to hear from all of 
you.) 


Leadership 


Providing direction for the team. (Let’s link ‘carbon dioxide’ to ‘producer’ .) 


Communication 1 


The clear and accurate exchange of information. (Yes, No) 



’We operationalized communication as the sum of all “Yes” and “No” messages, plus the 
total of all other messages, less the number of error messages. 



Measures 

The measures used in this study focused on team skills within a group concept mapping 
context. Performance was measured by how well the team’s concept map compared to experts’ 
maps. Process measures included team processes (as measured by message usage). Each measure 
is described below. 

Team outcome measures. Our team outcome measures were computed by comparing a 
teams’ concept map against a set of experts (Herl et al., 1996). Two semantic content scores, a 
concept map structure score, and the number of concepts and links in the group map were used as 
outcome measures. 

To derive semantic content scores, two different scoring criteria were used. The first 
technique used a stringent criteria. Every proposition in the group’s map (i.e., concept-link- 
concept tuple) was compared against all propositions in an expert’s map. A point was awarded for 
each match. This comparison was done against four expert maps, and the group’s semantic 
content score was the average score. A less stringent score was computed using link categories. 
Links in both the group and expert maps were categorized into the abstract categories. After this 
recoding, the score was recomputed. The effect of the categorizing specific links into more 
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abstract categories is to remove subtle differences between links (e.g., “causes” and “influenced” 
were classified as belonging to the ‘causal’ category). An organizational structure score was 
computed that measured the similarity in network characteristics between a group’s map and the 
experts’ maps. Herl et al. (1996) gives a complete discussion of all scoring algorithms. 

Procedure 

Each participant was randomly assigned to a team and a role (leader or member). There were 
no computer lab facilities at the school site so all data collection took place in classrooms that had 
ongoing instruction. Participants were arrayed in the back of the classroom to minimize the 
interaction with other students. The experimenters first introduced themselves and the study, then 
trained the students on the computer system and task, and then started the group concept mapping 
task. The training session took 13 minutes and the actual task 36 minutes. Leadership rotated 
every 6 minutes giving each participant the opportunity to control the concept map twice. 

Results 

Individual-Level Measures 

Each of the 37 messages were used at some point. Table 3 lists the frequency counts for 
each message. As Table 3 shows, although all messages were used, individual message usage 
varied from 4 (message #15 - “What if we change [L] to [L]?”) to 206 (message #34 - “No”). 
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Table 3. Frequency Count of Messages 



Msg. 

Number 


Freq. 

Count 


Msg. 

Number 


Freq. 

Count 


Msg. 

Number 


Freq. 

Count 


1 


60 


12 


47 


25 


16 


2 


41 


13 


13 


26 


27 


3 


47 


14 


9 


27 


23 


4(adapt)‘ 


31 


15 


4 


28 


50 


4(dec)‘ 


23 


16 


34 


29 


47 


4(lead)‘ 


14 


17 


65 


30 


54 


5 


30 


18 


24 


31 


108 


6 


91 


19 


47 


32 


112 


7 


18 


20 


31 


33 


167 


8 


46 


21 


36 


34 


206 


9 


26 


22 


82 


35 


46 


10 


23 


23 


58 






11 


13 


24 


30 







‘Message 4 was presented to participants as a combination of three message types: 
adaptability, decision making, and leadership. The category of the message depended 
on die specific message selected. 



Table 4 gives the frequency counts for each team process category. Interesting, decision 
making was used far less than the other processes. The frequency counts of the other categories 
show fairly uniform usage. These results contrast with the data from Pilot study 1 , which showed 
varied overall usage by category (adaptability - 26%, coordination - 2%, decision making - 14%, 
interpersonal - 3%, leadership - 38%). 
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Table 4. Frequency Count of Process Category 



Team process 


Total (pet 1 ) 


Mean 


SD 


Min. 


Max. 


Adaptability 

(Msg #: 4(adapt), 5, 7, 9, 29, 30, 31) 


314(17%) 


6.98 


4.59 


1 


18 


Coordination 

(Msg #: 13, 17, 18, 19, 20, 21, 22) 


298 (17%) 


6.62 


5.45 


0 


22 


Decision making 

(Msg #: 4 (dec), 10, 11, 14, 15, 16) 


106 (6%) 


2.36 


2.85 


0 


15 


Interpersonal 

(Msg #: 23, 24, 25, 26, 27, 28, 32) 


316(18%) 


7.02 


5.09 


0 


23 


Leadership 

(Msg #: 1, 2, 3, 4(lead), 6, 8, 12) 


346(19%) 


7.69 


5.27 


0 


18 


Yes or No 
(Msg #: 33,34) 


373 (21%) 


7.69 


5.27 


0 


18 


Error 

(Msg #: 35) 


47 (3%) 


1.04 


1.51 


0 


7 



'Percentages total to 101% due to rounding errors. 



For each participant, individual scores were calculated for the adaptability, coordination, 
decision making, interpersonal, leadership and teamwork processes. The scores were computed 
by counting the number of messages belonging to each category. In addition, a communication 
score was computed by summing each of the five teamwork process scores for each individual, 
plus the number of “Yes” and “No” messages sent, less the total number of error messages for that 
individual. An individual’s scale score was not adjusted for error messages. Table 5 gives means 
and standard deviations for each scale. As shown in Table 5, the average participant sent 
approximately 16% of the messages in each category. 
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Table 5. Individual-Level Teamwork Process Scales (N= 45) 



Process 


Mean 


SD 


Min 


Max 


Adaptability 


0.16 


0.12 


0.01 


0.51 


Coordination 


0.16 


0.14 


0.00 


0.59 


Decision making 1 


0.13 


0.18 


0.00 


0.81 


Interpersonal 


0.16 


0.13 


0.00 


0.55 


Leadership 


0.16 


0.12 


0.00 


0.42 


Communication 


0.78 


0.35 


0.15 


1.85 



’The decision making category contains 6 messages. Adaptability, coordination, 
interpersonal, and leadership contain 7 messages. Communication contains two 
messages, plus the composite of all other categories. 



Team-Level Process Measures 

Each of the individual-level teamwork process measures was used to calculate team-level 
scores. The individual scores for each process were summed among the three members of each 
team to generate a team score. Table 6 gives the means, standard deviations, and intercorrelations 
for each team-level teamwork process measure. Communication is a composite (simple sum) of 
the other five team process measures, plus the number of “Yes” and “No” messages sent, less the 
number of error messages sent. 



Table 6. Descriptives and Nonparametric (Spearman) Intercorrelations for Team-Level Teamwork 
Process Measures (N=15). 



Process 


Mean 


SD 


2 


3 


4 


5 


6 


1. Adaptability 


.47 


.25 


-.38 


-.50* 


-.20 


.24 


-.18 


2. Coordination 


.47 


.30 




.40 


.59** 


-.24 


.73** 


3. Decision making 


.40 


.30 






.18 


-.32 


.52* 


4. Interpersonal 


.47 


.25 








.15 


.77** 


5. Leadership 


.47 


.22 










.03 


6. Communication 


2.33 


0.60 













* p < .05. ** p < .01. 
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Team-Level Outcome Measures 

The concept mapping task generated a total of five team-level outcome measures: semantic 
content score (stringent), semantic content score (categorized), organizational structure score, 
number of terms used, and number of links used. Table 7 gives the descriptive statistics for each 
measure. 



Table 7. Descriptives Statistics for Team-Level Outcome Measures (N_= 15) 



Outcome measure 


Mean 


SD 


Min 


Max 


Semantic content score (strict) 


3.05 


2.15 


0.00 


7.50 


Semantic content score (categorized) 


4.35 


2.62 


0.75 


9.50 


Organizational structure score 


.19 


.07 


.10 


.30 


Number of terms used 


13.87 


3.96 


7 


18 


Number of links used 


17.00 


5.95 


8 


26 



As Table 7 shows, group maps were considerably lower than the expert criterion maps 
(stringent, M= 19.25, SD= 1.5; categorized, M= 22.5, S£>=21.5). This result is consistent with 
our pilot data and our assumption that our participants were content novices. This results also 
suggests that the Herl et al.’s (1996) scoring technique can discriminate between expert and 
novice concept maps. 
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Table 8. Nonparametric (Spearman) Correlations Between Team Processes and Outcome 
Measures (N = 15) 



Team process 


Semantic content score 
Stringent Categorized 


Organizational 
structure score 


No. of 
terms used 


No. of 
links used 


Adaptability 


.06 


-.11 


-.29 


-.41 


-.46* 


Coordination 


-.54* 


-.27 


-.32 


-.14 


-.10 


Decision making 


-.43 


-.26 


-.24 


-.08 


.07 


Interpersonal 


-.40 


-.24 


-.30 


-.21 


-.10 


Leadership 


-.04 


-.16 


-.08 


-.30 


-.22 


Communication 


. 72 ** 


-.52* 


-.56* 


-.40 


-.23 



* p < .05. ** p < .01. 



As Table 8 shows, the correlation between communication and the semantic content scores 
and structure score was statistically significant. This result suggests that the more messages a 
group sent, the lower their concept mapping performance. This is consistent with our pilot data, 
which also showed a significant negative correlation between communication and semantic 
content score. What is curious is that the did not find a relationship between decision making 
and the semantic content scores in particular. Our pilot data show a positive relationship, as did 
our past work O’Neil et al. (in press-b). 



Discussion 

Our findings suggest that real-time measurement of teamwork skills is feasible. Clearly, 
students using our networked concept mapping system were able to jointly construct a concept 
map. The number of concepts used by groups ranged from 7 to 18 (maximum possible), and the 
number of links ranged from 8 to 26. Some groups had sparse maps while others had fairly 
complex maps. There was nothing in the data, observed by the experimenters, or reported during 
debriefing sessions that participants had difficulty grasping the notion of concept mapping or that 
they had difficulty (procedurally) making concept maps. Participants in our pilot studies did, 
however, consistently comment that they did not like the use of predefined messages. Participants 
reported that they felt the messages were too constrained and they wanted to type their own 
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messages. 

In terms of team performance, our results are consistent with our assumptions about 
participants. We assumed that participants had little content knowledge, and participants’ semantic 
content scores were much lower than our experts. This suggests that Herl at al.’s (1996) concept 
map scoring algorithm can discriminate between experts and novices. 

Our findings of no significant correlations between most team processes and outcome 
measures were unexpected. In particular, we expected the use of decision making messages to 
play a critical role in our group concept mapping task. Our decision making messages were 
designed to give participants the opportunity to consider alternatives among different links and 
concepts (e.g., “What if add ‘producer’ - ‘requires’ - ‘carbon dioxide’?” or “We should think how 
‘producer’, ‘oxygen’, and ‘carbon dioxide’ relate to each other.”). We expected (as in O’Neil et 
al., in press-b) that the more teams used these kinds of messages the higher their team 
performance. 

A possible explanation for these findings is that our task is highly knowledge dependent, 
and thus our participants lacked the requisite knowledge to be able to engage each other at a 
substantive level. The use of messages may reflect more the procedural aspects of constructing a 
concept map instead of any substantive discussion about the content. Support for this is seen in 
the total number of message usage by all groups. Decision making messages, which we designed 
to allow participant to discuss the content, accounted for 6 % of messages use compared to 17%- 
2 1 % for the other categories . 

Another explanation is the split-attention effect (Sweller, 1994). While our task was fairly 
easy to carry out, the attentional demands were heavy. Selecting messages required participants to 
examine their message handout, drawing attention away from the map and messages that other 
members sent. Reading messages that other members sent draws attention away from focusing on 
the concept map, and constructing the map draws attention away from other members’ messages. 

If focusing on the concept map is the most important contributor to constructing good concept 
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maps, then it may have been the team leader who engaged others the least to have contributed the 
most. This is a reasonable assumption given the highly cognitive and knowledge driven nature of 
concept mapping. 

Support for the split-attention effect is seen in correlations between overall concept map 
activity (i.e., the number of times a concept map node or link was added, moved, deleted, or 
revised). Statistically significant nonparametric (Spearman) correlations of large magnitudes were 
found between concept map activity and every outcome measure: semantic content score (stringent) 
- ry(15)=.69,/?<.01; semantic content score (categorized) - r$(15)=.71,/><.01; organizational 
structure score - ry(15)=.68,/?<.01; number of terms used - r$(15)=.71,p<.01; and number of 
links used - ry(15)=.58,/?<.05. These findings suggest that the most important facet of group 
concept mapping is paying attention to the concept map. The overall level of activity may have 
induced too heavy a cognitive load on participants. 

These findings raise questions about our group concept mapping task relative to our 
teamwork process measures. In particular, what is a team and what is a teamwork task? A team is 
typically a group of individuals who (a) collectively share a common goal, and (b) are essential to 
the success of the team. The absence of an individual severely impacts the functioning of a team 
(McIntyre & Salas, 1995). The purpose of a team is to perform a task that cannot be performed by 
individuals alone. While this sometimes may be the goal in educational settings, more often the 
function of small groups in education is to improve individual learning (Webb & Palinscar, 1996). 
In our current work we have cast a teamwork perspective (as characterized by Tannenbaum et al., 
1996) on a weak teamwork task. Our collaborative concept mapping task may be more like a small 
group task than a teamwork task. A measurement system based on teamwork processes may be 
insensitive to small group processes, which may partially explain our findings. 

Despite these shortcomings we believe our approach to using networked computers to assess 
teamwork processes remains viable given the alternatives. The goal of large scale assessment of 
teamwork skills - let alone small scale ones - remains elusive. Existing approaches are labor and 
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time intensive, often involving hundreds of hours of transcription and coding. These approaches 
have little practical value as an assessment option, being untimely and prohibitively expensive. 

Our software architecture is designed to be domain independent and should transfer to other 
computer-based team environments. Such a measurement system would offer the capability of 
quickly assessing team processes and outcomes in educational environments (K-12) or industrial 
or military training environments. 
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